1. Field of the Invention
The present invention relates to an image processing method, image processing device, and recording medium that increase resolution, in an image capturing apparatus obtaining a plurality of pieces of image data.
2. Description of the Related Art
Since camera focus positions are adjusted during image capturing, when image capturing is performed by erroneous focusing, it is necessary to perform image capturing again. In addition, when it is desired to obtain image data focused on each of a plurality of objects having different perspectives, it is necessary to perform image capturing by focusing for each object.
In recent years, techniques called light field photography have been developed that obtains image data from multiple viewpoints by adding a new optical element in an optical system, and adjusts a focus position by image processing.
According to these techniques, since focus adjustment can be performed after image capturing, there is an advantage of being able to compensate the failure of focus adjustment during image capturing by image processing. Furthermore, by changing the image processing method, there are advantages of being able to obtain image data focused on any object in an image, from one piece of captured image data, and to reduce the number of times of image capturing.
In light field photography, the directions and intensities (light field, hereinafter abbreviated to LF) of light beams passing through a certain position, of a plurality of positions in space are obtained from multi-viewpoint images. Then, an image when formed on a virtual sensor is calculated by using the information of LF. Mathematical properties, mathematical bases and the like related to LF are discussed by R. NG, (“Light Field Photography with a Hand-held Plenoptic Camera” by R. NG, M. Levoy, M. Bredif, G. Duval, M. Horowitz, P. Hanrahan, Stanford Tech Report CTSR 2005-02, 2005).
By setting the position of this virtual sensor as appropriate, it is possible to perform the above-described focus adjustment after image capturing. Hereinafter, the processing for calculating an image that a virtual sensor obtains from multi-viewpoint images is referred to as reconstruction processing.
As image capturing apparatuses for obtaining an LF, plenoptic cameras in which a microlens array is placed behind a main lens, and camera arrays in which small cameras are arrayed have been known. An image when a sensor is virtually placed can be reconstructed after image capturing, from an obtained LF in any image capturing apparatus. At this time, it is possible to adjust, after image capturing, also the depth of field by changing the range of the LF to be reconstructed.
As methods for reconstructing an image on a virtual sensor from an LF, methods for projecting and converting a plurality of pieces of obtained image data on a virtual sensor to add and average the resultant data have been known (WO 2008/050904).
In addition, other than LF, as methods for obtaining high-resolution image data from a plurality of pieces of image data obtained by one camera, techniques that synthesize a plurality of pieces of pixel-shifted image data and increase resolution have been known (“Super Resolution Processing by Plural Number of Lower Resolution Images” by Shin Aoki, Ricoh Technical Report No. 24, November, 1998).
However, in the method of WO 2008/050904, aliasing due to the sampling of image capturing elements is generated in an image after reconstruction. In the method of WO 2008/050904, although sampling intervals are optionally finely set in order to increase resolution, there is a problem that substantial output resolution becomes 1/the number of angle divisions by aliasing. In order to increase output resolution, it is necessary to cancel aliasing.
In addition, in the method of “Super Resolution Processing by Plural Number of Lower Resolution Images” by Shin Aoki, Ricoh Technical Report No. 24, November, 1998, the method for generating one piece of image data in which aliasing is canceled relative to a plurality of pieces of obtained image data is described. However, this technique cannot handle the case where there is a desire to bring any object into focus after image capturing.